Object tracking device and a control method for object tracking device

ABSTRACT

In an object tracking device that obtains image data, detects an object, and tracks a target object, a matching unit collates image data of a characteristic model that represents a tracking object, which is a tracking target, with input image data, and outputs a candidate region of the tracking object. An object region determination unit determines an object region from the candidate region that is output by the matching unit. A depth map calculation unit calculates depth information relating to the input image. An object depth setting unit obtains the object region that has been determined by the object region determination unit in the past and the depth information that has been calculated by the depth map calculation unit, and sets a predetermined depth range where the object can exist. The object region determination unit determines an object region relating to the tracking object based on a region corresponding to the depth in a predetermined range set by the object depth setting unit and the candidate region that is extracted by the matching unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing technique thatoptically tracks an object.

Description of the Related Art

A technique that extracts a specific object image from images suppliedin time series and tracks the object is utilized for specifying a humanface region and a human body region in moving images. The objecttracking technique can be used in many fields, for example,teleconferencing, man-machine interfaces, security, monitoring systemsfor tracking any object, image compression, and the like.

In digital still cameras and digital video cameras, there has beenproposed a technique that extracts and tracks an object image in acaptured image that is specified by an operation using a touch panel andthe like, and consequently optimizes a focus state and an exposure statefor the object. Japanese Patent Application Laid-Open Publication No.2005-318554 discloses an image capturing apparatus that detects(extracts) and tracks the position of a face region included in acaptured image, focuses on the face, and captures the image with optimalexposure. Japanese Patent Application Laid-Open Publication No.2001-60269 discloses a technique that automatically tracks a specificobject by using template matching. In the template matching processing,a partial image obtained by cutting out an image region including aspecific object image (hereinafter, also referred to as a “trackingtarget”) is registered as a template image. With the use of an inputinterface such as a touch panel, any region included in the image isspecified, and the template image is registered as serving the regionserving as a reference. A specific object can be tracked by calculatingan area that is the highest in similarity or an area that is the lowestin dissimilarity in the image by comparison with the template image.

In the template matching processing, the similarity of pixel patterns isused as an evaluation scale. Accordingly, if the pixel patterns in thepartial region in the tracking target and the objects other than thetracking target (for example, a background) are similar to each other,an object that should not be tracked may be tracked. As another trackingmethod, there is a method in which color histogram similarity isutilized as the evaluation scale for matching. In this case, if theproportions of the color in the partial region between the trackingtarget and the objects other than the tracking target are similar toeach other, an object that should not be tracked may be tracked. Inorder to improve an accuracy of the object tracking, new information fordistinguishing the tracking target from the objects other than thetracking target is necessary.

SUMMARY OF THE INVENTION

The present invention improves the accuracy of object tracking by usingdistance information (depth information) relating to an object astracking information.

A device according to the present invention is an object tracking devicethat obtains image data, detects an object, and tracks an object to betargeted comprising: a matching unit that is configured to collate imagedata of a tracking object that is a tracking target with image data thathas been obtained, and output information about a candidate region ofthe tracking object; a region determination unit that is configured todetermine an object region relating to the tracking object from thecandidate region that is output by the matching unit; a depthinformation calculation unit that is configured to calculate depthinformation for the object; and a depth range setting unit that isconfigured to obtain information about the object region that has beendetermined by the region determination unit at a previous time earlierthan the current time and the depth information that has been calculatedby the depth information calculation unit, and set a depth range on thebasis of a depth of the tracking object, wherein the regiondetermination unit determines an object region relating to the trackingobject based on an image region falling within the depth range set bythe depth range setting unit and a candidate region that is output bythe matching unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animaging apparatus according to an embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating a pixel arrangement in theembodiment of the present invention.

FIG. 3A is a schematic plan diagram illustrating pixels in theembodiment of the present invention.

FIG. 3B is a schematic cross-sectional diagram illustrating the pixelsin the embodiment of the present invention.

FIG. 4 is a schematic diagram illustrating a relation between the pixelsand a pupil division in the embodiment of the present invention.

FIG. 5 is a schematic explanatory diagram of an imaging element and thepupil division in the embodiment of the present invention.

FIG. 6 is a block diagram illustrating a configuration example of anobject tracking device according to the embodiment of the presentinvention.

FIG. 7A and FIG. 7B illustrate template matching according to theembodiment of the present invention.

FIG. 8 is a flowchart illustrating object tracking processing accordingto the embodiment of the present invention.

FIG. 9A to FIG. 9D illustrate a specific example of a setting distancerange according to the embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, a description will be given of each preferred embodiment ofthe present invention with reference to the accompanying drawings. Anoptical apparatus according to the present invention is applicable tovarious types of lens apparatuses and imaging apparatuses, binoculars,and the like. As preferred embodiments of the present invention,although the following embodiments show an example of application toimaging apparatuses, for example, digital still cameras and videocameras, they are not intended to limit the technical scope of thepresent invention.

(First Embodiment)

With reference to FIG. 1, a description will be given of a configurationof an imaging apparatus according to a first embodiment of the presentinvention. FIG. 1 is a block diagram illustrating a configurationexample of an image capturing apparatus 100. The image capturingapparatus 100 performs processing that captures an object and recordsdata of moving images and still images on a recording medium. The datacan be recorded on various media, as recording mediums, including atape-like recording medium, a solid memory, an optical disk, a magneticdisk, and the like. Each unit in the image capturing apparatus 100 isconnected via a bus 160 and is controlled by a CPU (Central ProcessingUnit) 151.

An image shooting lens (lens unit) 101 includes a fixed first group lens102, a zoom lens 111, a diaphragm 103, a fixed third group lens 121, anda focus lens 131. A diaphragm control unit 105 drives the diaphragm 103via a diaphragm motor 104 in accordance with a command of a CPU 151,adjusts an aperture diameter of the diaphragm 103, and performs theadjustment of a light amount during shooting. A zoom control unit 113drives the zoom lens 111 via a zoom motor 112 and changes the focaldistance. Additionally, a focus control unit 133 determines a driveamount of the focus motor 132 depending on a deviation amount of thefocal position based on a focus detection signal of the image shootinglens 101, and performs drive control of the focus motor 132. Themovement of the focus lens 131 is controlled by the focus control unit133 and the focus motor 132, and thereby AF (automatic focus adjustment)control is realized. In FIG. 1, although the focus lens 131 is simplyillustrated as a single lens, it is usually configured of a plurality oflenses.

Light from an object is forms an image on an imaging element 141 viaeach optical member that configures the image shooting lens 101. Theobject image formed on the imaging element 141 is converted into anelectric signal by the imaging element 141. The imaging element 141 is aphotoelectric conversion element that photoelectrically converts anobject image (optical image) into an electric signal. The imagingelement 141 is configured by a plurality of microlenses and a pixel unitthat includes a plurality of photoelectric conversion elementscorresponding to each microlens. That is, a parallax image can begenerated by using the imaging element including the plurality ofphotoelectric conversion elements (first photoelectric conversionelement and the second photoelectric conversion element) that share onemicrolens. The image capturing apparatus 100 includes an object trackingdevice that tracks a specific object, and performs calculation by usingdistance information (depth information) calculated based on theparallax image in the object tracking.

With reference to FIG. 2, a description will be given of an arrangementof imaging pixels and focus detection pixels of the imaging element 141in the present embodiment. FIG. 2 illustrates an imaging pixelarrangement of a two-dimensional CMOS (Complementary Metal OxideSemiconductor) sensor in the range of 4 columns×4 rows and illustrates afocus detection pixel arrangement thereof in the range of 8 columns×4rows. A pixel group 200 of 2 columns×2 rows includes a pair of pixel200R, 200G, and 200B shown below:

-   -   Pixel 200R (see the upper left position): pixels having spectral        sensitivity to R (red)    -   Pixel 200G (see the upper right and lower left positions):        pixels having spectral sensitivity to G (green)    -   Pixel 200B (see the lower right position): pixels having        spectral sensitivity to B (blue)        Each pixel unit is configured of a first focus detection pixel        201 and a second focus detection pixel 202, which are arranged        in 2 columns×1 row. A large number of pixels of 4 columns×4 rows        (focus detection pixels of 8 columns×4 rows) shown in FIG. 2 is        arranged in a grid shape on a plane, and consequently, the        captured image signals and focus detection signals can be        obtained.

A plan diagram in which one pixel 200G in the imaging element shown inFIG. 2 is viewed from the light receiving surface side of the imagingelement (+z side) is shown in FIG. 3A. The z-axis is set in a directionperpendicular to the drawing of FIG. 3A, and the front side is definedas the positive direction of the z-axis. Additionally, the y-axis is setin the vertical direction orthogonal to the z-axis and the upper side isdefined as the positive direction of the y-axis, and the x-axis is setin the horizontal direction orthogonal to the z-axis and the right sideis defined as the positive direction of the x-axis. Along a-a cut linein FIG. 3A, a cross-sectional diagram when viewed from the −y directionis shown in FIG. 3B.

In the pixel 200G shown in FIG. 3B, a microlens 305 that collectsincident light on the light receiving surface side (+z-direction) ofeach pixel is formed, and a plurality of photoelectric conversion unitsthat have been divided is included. For example, the divided number inthe x-direction is defined as N_(H), and that in the y-direction isdefined as N_(V). In FIG. 3, an example in which the pupil region isdivided in two in the x-direction, that is, the case of N_(H)=2 andN_(V)=1 is illustrated, and a photoelectric conversion unit 301 and aphotoelectric conversion unit 302, which serve as the sub-pixels, areformed. The photoelectric conversion unit 301 corresponds to the firstfocus detection pixel 201, and the photoelectric conversion unit 302corresponds to the second focus detection pixel 202. The photoelectricconversion unit 301 and the photoelectric conversion unit 302 are formedas, for example, a pin structure photodiode in which an intrinsic layeris interposed between a p-type layer 300 and an n-type layer.Alternatively, as necessary, it may be possible to form them as a pnjunction photodiode, with the intrinsic layer omitted. In each pixel, acolor filter 306 is formed between the microlens 305, and thephotoelectric conversion unit 301 and the photoelectric conversion unit302. As necessary, it may be possible to change spectral transmittanceof the color filter 306 for each sub-pixel and possible to omit thecolor filter.

The light incident to the pixel 200G are collected by the microlens 305,and additionally, after being spectrally dispersed by the color filter306, they are received by the photoelectric conversion unit 301 and thephotoelectric conversion unit 302. In the photoelectric conversion unit301 and the photoelectric conversion unit 302, electrons and holes(positive holes) are pair-produced depending on the received lightamount, and after they are separated in a depletion layer, electronshaving negative charge are accumulated in an n-type layer (notillustrated). In contrast, the holes are discharged to the outside ofthe imaging element through a p-type layer connected to a constantvoltage source (not illustrated). The electrons accumulated in then-type layer (not illustrated) of the photoelectric conversion unit 301and the photoelectric conversion unit 302 are transferred to acapacitance unit (FD) via a transfer gate, and are converted intovoltage signals.

FIG. 4 is a schematic explanatory diagram illustrating thecorrespondence relation between a pixel structure and the pupildivision. FIG. 4 illustrates a cross-sectional diagram that illustratesa cut surface along a line a-a of the pixel structure shown in FIG. 3A,viewed from the +y-direction, and illustrates an exit pupil plane of animaging optical system (see an exit pupil 400), viewed from the −zdirection. In the cross-sectional diagram of FIG. 4, the x-axis andy-axis are illustrated by inverting the state shown in FIG. 3A, in orderto correspond with coordinate axes on the exit pupil plane. A firstpupil partial region 501 corresponding to the first focus detectionpixel 201 is almost in a neighboring relation by the microlens 305 withrespect to a light receiving surface of the photoelectric conversionunit 301, of which the center of mass deviates in the −x direction. Thatis, the first pupil partial region 501 represents a pupil region thatenables receiving light in the first focus detection pixel 201, and thecenter of mass deviates in the +x direction on the pupil plane.Additionally, a second pupil partial region 502 corresponding to thesecond focus detection pixel 202 is almost in a neighboring relation bythe microlens 305 with respect to a light receiving surface of thephotoelectric conversion unit 302 of which the center of mass deviatesin the +x direction. The second pupil partial region 502 represents apupil region that enables receiving light in the second focus detectionpixel 202, and on the pupil plane, the center of mass is displaced inthe −x direction.

A pupil region 500 shown in FIG. 4 is a pupil region in which light canbe received in the entire pixel 200G in a case where the photoelectricconversion unit 301 and the photoelectric conversion portion 302 (thefirst focus detection pixel 201 and the second focus detection pixel202) are both combined. The correspondence relation between the imagingelement and the pupil division is shown in the schematic diagram of FIG.5. Light fluxes that have passed through different pupil partialregions, the first pupil partial region 201 and the second pupil partialregion 502, are incident to each pixel of the image element at differentangles. Light incident to an imaging surface 800 is received by thefirst focus detection pixel 201 and the second focus detection pixel202, which are divided into N_(H) (=2)×N_(V) (=1). The photoelectricconversion unit 301 of the first focus detection pixel 201 and thephotoelectric conversion unit 302 of the second focus detection pixel202 convert lights into the electrical signals. In the presentembodiment, an example in which the pupil region is pupil-divided intotwo in a horizontal direction is described. If necessary, pupil divisionmay be performed in a perpendicular direction.

The imaging element 141 according to the present embodiment includes afirst focus detection pixel that receives a light flux passing throughthe first pupil partial region of the imaging optical system, and asecond focus detection pixel that receives a light flux passing throughthe second pupil partial region of the imaging optical system, which isdifferent from the first pupil partial region. A plurality of theimaging pixels that receive a light flux passing through the pupilregion obtained by combining the first pupil portion region and thesecond pupil partial region of the imaging optical system is arranged ina two-dimensional array state. That is, each imaging pixel is composedof the first focus detection pixel and the second focus detection pixel.If necessary, a configuration may be adopted in which the imaging pixel,the first focus detection pixel, and the second focus detection pixelserve as a separate pixel configuration, and the first focus detectionpixel and the second focus detection pixel are partially distributed inthe imaging pixel arrangement.

In the present embodiment, “A-image”, which is the first focus detectionsignal, is generated by collecting light receiving signals of the firstfocus detection pixel 201 in each pixel, and “B-image”, which is thesecond focus detection signal, is generated by collecting lightreceiving signals of the second focus detection pixel 202 in each pixelin the imaging element 141. An object tracking unit 161 described belowcalculates an image deviation amount based on the A-image and B-imagehaving parallax, and the processing that calculates the distanceinformation (depth information) from the image deviation amount isperformed. Additionally, for each pixel of the imaging element 141, theA-image and the B-image are add to generate an “A+B image”, and imagedata used for display or recording can be generated. The image signalthat has been generated by focusing (an image) on the imaging element141 and performing photoelectric conversion is output to an imagingsignal processing unit 142 in FIG. 1. The imaging signal processing unit142 processes the image signal that is input, and performs shapingprocessing to the image data.

The imaging signal processing unit 142 outputs the image data that hasbeen processed to an imaging control unit 143. The image data that hasbeen processed is temporarily stored and accumulated in a RAM (randomaccess memory) 154. After an image compression extension unit 153performs the compression processing on the image data accumulated in theRAM 154, it performs a process that records the data on an imagerecording medium 157. In parallel with this, the image data accumulatedin the RAM 154 is transmitted to an image processing unit 152. The imageprocessing unit 152 processes the image data, for example, theprocessing that reduces or enlarges the data to an optimal size. Theimage data that has been processed to the optimal size is transmitted toa monitor display 150 and the image is displayed. An operator canobserve the shot images in real time while viewing the image displayedon the monitor display 150. Note that, immediately after the shooting,the operator can check the shot image by displaying it on the monitordisplay 150 for a predetermined period of time. An operation unit 156includes various operation switches, and is used when the operatorperforms an instruction on the image capturing apparatus 100. Theoperation instruction signal input from the operation unit 156 istransmitted to the CPU 151 via the bus 160.

The CPU 151 determines setting values of various parameters based on theoperation instruction signal input from the operation unit 156 or thesize of the pixel signal of the image data that is temporary accumulatedin the RAM 154. The various parameters are, for example, an accumulationtime of the imaging element 141, and a gain setting value when an outputfrom the imaging element 141 to the imaging signal processing unit 142is performed. The imaging control unit 143 obtains a command signalabout the accumulation time and the gain setting value from the CPU 151,and controls the imaging element 141 in accordance with the commandsignal.

The data of the A+B image, which is image data accumulated in the RAM154, is also transmitted to the object tracking unit 161. The objecttracking unit 161 tracks a specific object by using a plurality of imagedata having different image capture times. As a result for the tracking,a partial region (image region) that shows the specific object isextracted. Additionally, each data of the A-image and the B-imagecorresponding to the parallax image is also accumulated in the RAM 154.Each data of the A-image and the B-image is used as information forcalculating distance information (depth information) based on theparallax image, and for tracking the specific object. The details willbe described below.

The output of the object tracking unit 161 is reported to eachprocessing unit via the bus 160. For example, the focus control unit 133obtains the output of the object tracking unit 161 and performs AFcontrol on the specific object region. Additionally, the diaphragmcontrol unit 105 obtains the output of the object tracking unit 161 andperforms exposure control using a luminance value on the specific objectregion. The image processing unit 152 obtains the output of the objecttracking unit 161 and performs gamma correction, white balanceprocessing, and the like, based on the specific object region.Additionally, the monitor display 150 displays an object regionincluding a part of the object image, which is a tracking target, indistinction from other image regions by using a rectangular frame andthe like, in accordance with the output of the object tracking unit 161.

A power management unit 158 manages a battery 159 and supplies a stablepower to the entire image capturing apparatus 100. A flash memory 155stores a control program necessary for the operation of the imagecapturing apparatus 100. When the operator performs a start operation ofthe image capturing apparatus 100, the power OFF state transits to thepower ON state, and the control program stored in the flash memory 155is loaded and read in a part of the RAM 154. The CPU 151 controls theoperation of the image capturing apparatus 100 in accordance with thecontrol program that has been loaded in the RAM 154.

Next, with reference to FIG. 6, a description will be given of thedetail of the object tracking unit 161. The object tracking unit 161detects an object to be tracked, and tracks a specific object by usingimage data supplied sequentially. As a result for the tracking, apartial region representing the specific object in the image isextracted. The object tracking unit 161 utilizes the image data of theA+B image in detection processing and the matching processing fortracking, and additionally, performs object tracking with high accuracyby using the distance information about the object. In order tocalculate the distance information, the object tracking unit 161utilizes the data of the A-image and the B-image.

FIG. 6 is a block diagram illustrating a configuration example of theobject tracking unit 161. An object detection unit 601 detects apredetermined object image to be targeted from the A+B image, which isan input image, and serves the image as a tracking target in objecttracking. For example, in case of face detection, the object trackingunit 161 specifies a human face region and the like as the objectregion. As face detection techniques, for example, there are a method inwhich knowledge regarding a face (skin color information, shapeinformation such as eyes, nose and mouth) is utilized, and a method inwhich a discrimination processing unit for face detection is configuredby learning algorithm typified by a neural network. Additionally, inface detection, in order to improve a recognition rate, face recognitionis commonly performed by combining a plurality of methods.

Specifically, there is a method in which face detection is performed byutilizing a wavelet transform and a feature value of images (seeJapanese Patent Application Laid-Open Publication No. 2002-251380) andthe like. Alternatively, a configuration may be used in which, in a formin which the operation unit 156 includes an input interface unitincluding a touch panel and an operation button, the operator specifiesany object image included in a captured image as a tracking target. Inthis case, the object detection unit 601 obtains positional informationspecified by the operation unit 156, and detects the object region basedon the positional information.

A matching unit 602 in FIG. 6 obtains data of the A+B image, which is aninput image, and registers the object region that has been detected bythe object detection unit 601 as a template. The matching unit 602performs matching processing that collates the template that has beenregistered with the partial regions of the input image that aresequentially input, and outputs a plurality of higher evaluation valuesand region information as a candidate region of the tracking object.There are a number of matching methods, and in the present embodiment, amethod by template matching based on the differences between pixelpatterns is applied. With reference to FIG. 7, a description will begiven of the detail of the template matching.

FIG. 7A illustrates an object model (template) in the template matching.An image 701 on the left represents an image in an object region that isa tracking target. A description will be given of an example in whichthe pixel pattern of the image 701 serves as the feature value. Dataarrangement 702 represents a feature value of the image 701, and atwo-dimensional arrangement in a case where a luminance signal of thepixel data serves as a feature value is illustrated. Two-dimensionalcoordinates (i, j) are set in the template region, wherein the variable“I” represents position coordinates in the horizontal direction, and thevariable “j” represents position coordinates in the vertical direction.A feature value in the two-dimensional coordinates (i, j) is denoted by“T (i, j)”, the number of horizontal pixels is denoted by “W”, and thenumber of vertical pixels is denoted by “H”. The feature value T (i, j)is represented by the following formula.T(i,j)={T(0,0),T(1,0), . . . ,T(W−1,H−1)}  [Formula 1]

FIG. 7B illustrates a search image when searching a tracking target. Animage 703 on the left shows an image in a range for performing thematching process. In the two-dimensional coordinates in the searchimage, the horizontal direction is defined as the x-direction, and thevertical direction is defined as the y-direction, which are representedby (x, y). A rectangular partial region 704 shown in the image 703 is apartial region for obtaining an evaluation value of the match. A featurevalue 705 of the partial region 704 is represented by a two-dimensionalarrangement, and the luminance signal of the image data serves as thefeature value similar to the case of template. The feature value in thetwo-dimensional coordinates (i, j) in the partial region 704 is denotedby S (i, j), the number of horizontal pixels is denoted by W, and thenumber of vertical pixels is denoted by H. The feature value S (i, j) isrepresented by the following formula.S(i,j)={S(0,0),S(1,0), . . . ,S(W−1,H−1)}  [Formula 2]

In the present embodiment, as a calculation method for evaluating thesimilarity between the template region and the partial region, the sumof the absolute values of the difference, what is referred to as “SAD(Sum of Absolute Difference)” is used. When the SAD value is representedby V (x, y), this is calculated by the following formula.

$\begin{matrix}{{V\left( {x,y} \right)} = {\sum\limits_{j = 0}^{H - 1}\;{\sum\limits_{i = 0}^{W - 1}\;{{{T\left( {i,j} \right)} - {S\left( {i,j} \right)}}}}}} & \left\lbrack {{Formula}\mspace{14mu} 3} \right\rbrack\end{matrix}$

While shifting the partial region 704 by each pixel in order from theupper left of the image 703 in the search region, the calculation of theSAD value V (x, y) is performed. If the SAD value V (x, y) obtained bythe calculation is a minimum value, the coordinates (x, y) indicate aposition that is most similar to the template. That is, the positionwhere the SAD value shows a minimum value is a position having a highprobability that the tracking target exists in the search image.

In the above description, although an example in which one type ofinformation, which is a luminance signal, is used as the feature valueis shown, it may be possible that two or more pieces of informationabout, for example, the signals of the brightness, hue, and saturation,or the signals combing them, are handled as the feature value.Additionally, as a calculation method for the matching evaluation value,although a description was given by using the SAD value as an example, acalculation method, for example, normalization cross-correlation, whatis referred to as “NCC Normalized Correlation Coefficient)”, and thelike may also be used. Additionally, the present invention is notlimited to only template machining to apply the present invention, andother matching methods, for example, histogram matching based on thesimilarity of the histogram may be adopted.

An object region determination unit 603 in FIG. 6 determines one objectregion from among the candidate regions of the object to be tracked,based on the plurality of higher evaluation values and the regioninformation supplied from the matching unit 602. For example, a regionhaving a highest evaluation value is determined to be the object region.This is a simple determination method, but if pixel patterns are similarbetween the tracking target and a partial region in a background oranother object, an evaluation value of the object region not to betracked may increase. Accordingly, in the present embodiment, byreferring to the distance information (distance map data, depthdistribution) calculated based on the parallax image data, theprocessing that determines a correct object region is performed. Thus,the probability of detecting an object region to be tracked can beincreased. Information about the one object region that has beendetermined by the object region determination unit 603 is output fromthe object tracking unit 161.

A distance map calculation unit 604 obtains the data for a pair ofimages (A-image, B-image) having a parallax in the horizontal direction,and calculates the object distance. The object distance is informationindicating a distance (depth) from the image capturing apparatus to theobject. The image deviation amount can be detected by performingcorrelation calculation processing on a pair of images having parallaxin the horizontal direction. In the detection processing of the imagedeviation amount, for example, correlation calculation is performed foreach small block obtained by dividing the image region into smallregions (see Japanese Patent Application Laid-Open Publication No.2008-15754). By multiplying a predetermined conversion coefficient withrespect to the image deviation amount calculated by the correlationcalculation, a deviation (defocus amount) in the imaging plane of theimage is calculated. Hereinafter, the defocus amount that has beencalculated is referred to as “calculated distance”. The informationabout the distance distribution in which the calculated distance isassigned to each pixel of the image on the imaging surface is referredto as a “distance map”.

An object distance setting unit 605 obtains information about the objectregion that has been determined by the object region determination unit603 at a past timing and the distance map data that has been calculatedby the distance map calculation unit 604, and sets a distance in apredetermined range in which the object can exist. The object regiondetermination unit 603 determines one region that corresponds to thedistance in the predetermined range that is set by the object distancesetting unit 605 and is a candidate region that has been extracted bythe matching unit 602 as the object region. Specifically, a variablerepresenting the passage of time is denoted by “n”, and the image dataat time n−1 and the image data at time n has been obtained. In thiscase, time n−1 and time n are continuous, and the object distancebetween the two times does not change significantly. The object distancesetting unit 605 calculates the distance at which the object exists attime n−1, based on the partial region that has been determined by theobject region determination unit 603 at time n−1 (object region) and thedistance map at time n−1 in the partial region. The object distancesetting unit 605 sets the distance range (depth range. hereinafter,referred to as a “setting distance range”) on the basis of thecalculated distance of the target object, and the partial region thatcorresponds to within the setting distance range is determined by thedistance map at time n. The object region determination unit 603extracts only the partial region corresponding to within the settingdistance range, with respect to the plurality of higher evaluationvalues by the matching unit 602 at time n. Subsequently, the objectregion is determined from the partial region that has been extracted.

Next, with reference to a flowchart in FIG. 8, a description will begiven of the process performed by the object tracking unit 161. First,the captured image at time n−1 is supplied as an input image to theobject tracking unit 161 (S801). The object detection unit 601 detects aspecific object region from this image (S802). The matching unit 602registers a template, which is an object model of the template matching,based on the specific object region that has been detected (S803).Additionally, at the initial timing of the tracking processing, theobject distance setting unit 605 initializes and clears a value of theobject distance (setting distance range, setting depth range) (S804).Note that S802, S803, and S804 are performed in a random order.

Next, the captured image at a time n that differs from the time in S801is supplied as an input image (S805). The input image in S805 representsa search image by the object tracking unit 161. Based on this searchimage, the matching unit 602 performs collation by template matching(S806). Additionally, the distance map calculation unit 604 calculatesthe distance map based on the data of the input image in S805 (S807).Note that S806 and S807 are performed in random order.

In S808, the object region determination unit 603 narrows down thematching result in S806, based on the distance range that has been setin advance (the setting distance range before being updated in S812) andthe distance map that has been calculated in S807. Regarding the initialprocess, however, because the setting distance range does not exist(because the range is cleared in S804), the matching result is nevernarrowed down. Next, the object region determination unit 603 determinesa partial region in which the highest evaluation value is obtained asthe result for the matching that has been narrowed down in the settingdistance range, as an object region (S809). When one object region thatbelongs to the setting distance range is determined, the object trackingunit 161 determines whether or not it continues tracking based on theobject region that has been determined (S810). As an example of thedetermination, in S808, the object tracking unit 161 determines nocontinuance of tracking if all regions are not matched as the result formatching.

If no continuance of the object tracking processing is determined (“NO”is determined in S810), the object tracking processing ends. Forexample, the process ends if the object image to be tracked no longerexists in the image within the search range. In contrast, if thecontinuance of the object tracking processing is determined (“YES” isdetermined in S810), the process proceeds to S811.

In S811, the object region determination unit 603 updates the templateof the matching unit 602 based on the object region that has beendetermined. The object distance setting unit 605 updates the distancemap and updates the setting distance range based on the object regionthat has been determined (S812). The details of the setting distancerange will be described below. Next, the process returns to S805, andthe object tracking unit 161 continues the object tracking processingbased on the input images that are supplied sequentially.

In the above description, processing that narrows down the matchingevaluation value based on the setting distance range was exemplified.The present invention is not limited thereby, and the object region maybe determined by the two conditions, the matching evaluation value andthe setting distance range (setting depth range). For example, thepresent embodiment may be implemented by a structure in which the regionto be matched is restricted by using the setting distance range and thedistance map thereby to avoid the output of the matching evaluationvalue in the setting distance range.

As described above, the object tracking device according to the presentembodiment determines an object region from the region that correspondsto the distance in the predetermined range and from the candidate regionthat has been extracted by the matching unit 602. The distance (depth)in the predetermined range is set based on the result for the objecttracking, which is past history information. Appropriately utilizing thecondition of the distance in the predetermined range allows improvingthe accuracy of the object tracking. For example, if the settingdistance range is too large, the restriction based on the distanceinformation is reduced, and the effect of referring to the distanceinformation is thereby reduced. Additionally, if the setting distancerange is too small, the object moves significantly in the depthdirection (direction along the shooting direction), and it may be out ofthe setting distance range. In this case, there is a concern that theregion to be tracked is excluded from the candidate region. Accordingly,in the present embodiment, an optimum distance range is set in order toimprove the accuracy of the object tracking.

With reference to FIG. 9, a description will be given of a process thatsets a distance range depending on the condition of the object tracking.On the basis of the position shown in the image capturing apparatus 100in FIG. 9, a three-dimensional space coordinate system consisting of thex-axis, the y-axis, and the z-axis is set. The x-y plane consisting ofthe x-axis and the y-axis is a plane parallel to the imaging surface ofthe captured image, and the z-axis is an axis along the optical axisdirection in the imaging optical system orthogonal to the x-y plane. Thedistance (depth) is calculated on the basis of the position of the imagecapturing apparatus 100 in the z-axis direction. Among a plurality ofobjects 901 to 904, the object 901 represents a tracking target, and theobjects 902 to 904 represent objects that are different from thetracking target. As shown in FIG. 9, the objects 902 and 903 are locatedcloser to the image capturing apparatus 100 as compared with thetracking object 901, and the object 904 is located far from the imagingcapturing apparatus 100 as compared with the tracking object 901. Whenperforming the setting of the distance range, the object tracking devicemounted on the image capturing apparatus 100 uses history informationthat has been obtained at a previous time earlier than the current time.The history information is, for example, distance information about theobject region that has been obtained in a most recent predeterminedperiod of time. FIG. 9 shows a manner in which setting distance ranges905, 906, 907, and 913 in each tracking condition are set.

FIGS. 9A and 9B illustrate a case where the distance range is set basedon a past movement of the tracking target. FIG. 9A shows a case in whichthe tracking target moves slowly. The setting distance range 905 is setin a range having a width in the front or back of the position of thetracking object 901 in the z-axis direction. FIG. 9B shows a case inwhich the target moves fast. The setting distance range 906 is set to arange having a width in the front or back of the position of thetracking object 901 in the z-axis direction, and it is larger than thesetting distance range 905. That is, when comparing the setting distanceranges 905 and 906, the one in which the tracking target moves faster islarger in the setting distance range. In the case of the tracking targetthat moves fast, there is a probability that the tracking object will beexcluded from the setting distance range, and therefore the settingdistance range is made large. In contrast, in the case of the trackingtarget that moves slowly, the probability in which the tracked object isexcluded from the setting distance range is low. Hence, the settingdistance is made small in order to enhance the effect of referring tothe distance information.

FIGS. 9C and 9D illustrate a case in which the distance range is setbased on the distance relation (distance difference) between thetracking object and the object that is different from the trackingobject. In FIG. 9C, the distance range is set based on the distanceinformation about a plurality of objects other than the tracking target.The plurality of objects are the objects 902 and 903, which are thenearest neighbor, located frontwards closer to the image capturingapparatus 100 with respect to the tracking object 901, and an object904, which is the nearest neighbor, located backwards far from the imagecapturing apparatus 100. Setting of the distance range is performedbased on the distance information about the objects 902 and 903 and thatabout the object 904. In the object tracking, the purpose of referringto the distance information is to distinguish the tracking target fromanother object. If the distance between the tracking target and anotherobject is short (if the distance difference is small), the objecttracking device reduces the setting distance range. On the contrary, ifthe distance between the tracking target and another object is long (ifthe distance difference is large), the object tracking device increasesthe setting distance range.

FIG. 9D illustrates a case in which the distance range is set based onthe distance relation with the object that is similar to the target. Theobject that is similar to the tracking target indicates an object havingcharacteristics similar to those of the tracking target. The objects 902and 904 shown in FIG. 9D represent objects that are not similar to theobject 901, which is a tracking target. Additionally, the objects 908and 909 represent objects that are similar to the object 901, which is atracking target. For determining the similarity, matching evaluationvalues are used. That is, the “object is similar” means that thematching evaluation values between the tracking target and anther objectare close. When using the template matching method described in thepresent embodiment, a similar object indicates an object that is similarin pixel pattern to the image that has been obtained. Alternatively,when using the color histogram matching method, the similar object is anobject that is similar to the percentage of color. If distinguishingbetween the tracking target and another object is difficult only byusing the matching evaluation values, it is effective to refer to thedistance information about the object because distinguishing theplurality of objects can be allowed. That is, it is possible to improvethe accuracy of the object tracking with the use of the distanceinformation by setting the distance range based on the distance relation(distance difference) between the tracking target and the objectssimilar to the tracking target. Consequently, the setting distance rangecan be set relatively large, and the object tracking device can respondto the fast-motion of the tracking target.

Additionally, the object tracking device distinguishes the motion of theobject (the moving speed in the shooting direction) that is similar tothe tracking target. For example, the object tracking device determinesthat an object similar to the tracking target is a moving object or astationary object, and changes the setting of the distance range inaccordance with the determined result. The distance range is set smallif the object is a moving object, and the distance range is set large ifthe object is a stationary object. Alternatively, if the object similarto the tracking target is a moving object, the faster the object moves,the smaller the distance range is set in accordance with the movingspeed. As described above, in the distance range setting processingperformed by the object distance setting unit 605, the setting distancerange is dynamically changed in accordance with the situation of theobject tracking, and consequently in the object tracking processing, thedistance information can be effectively utilized.

In the present embodiment, even in a case in which the pixel patternsare similar in each image between the objects, which is a trackingtarget, and another object, and the distinction is difficult only by thematching evaluation values, the target can be accurately tracked byusing the distance information if distances to each object differ. Thatis, even if the tracking object and another object are similar in thepixel patterns and the color histogram, accurate object tracking ispossible if they differ in distance. According to the presentembodiment, in the optical tracking of the object, it is possible toimprove the accuracy of tracking by utilizing distance information aboutthe object.

(Second Embodiment)

Next, a description will be given of a second embodiment of the presentinvention. In the present embodiment, a countermeasure is taken againstthe lowering of a tracking accuracy when distance information isreferred to in a case in which the distance information is not correct.A device according to the present embodiment further includes acalculation unit that calculates a reliability map that indicates thereliability of the distance map data, together with the distance map.Specifically, in FIG. 6, the distance map calculation unit 604 isreplaced with a distance map and reliability map calculation unit 614.When the distance map and reliability map calculation unit 614 calculatethe distance information by the distance information calculationprocessing, it executes the reliability calculation processing andcalculate the reliability map relating to the distance information. Forexample, if the value of the reliability is small, the reliability ofthe corresponding distance map data is low.

Here, a description will now be described with respect to an example ofgenerating processing of the reliability map. In the calculationprocessing of the distance map, a pair of image regions having aparallax in the horizontal direction is divided into small regions,correlation calculation is performed for each of the small blocks, andan image deviation amount is consequently detected. In a case in whichthe correlation calculation is based on the similarity of the imagepattern, if the image pattern of the small block is an aggregate ofpixels that are similar each other, the peak value of the correlationoccurs with difficulty. Therefore, the detection of the correct imagedeviation amount is difficult. Accordingly, if the difference betweenthe average value and the peak value (the maximum value in the case ofsimilarity) in the calculation value obtained as the result for thecorrelation calculation is small, the reliability is determined to below. That is, the reliability can be defined by utilizing thedifference. This reliability has coordinate information so as to becalculated by each small block, and therefore a map (reliability map)representing the distribution of the reliability information about thedistance information is generated.

The object region determination unit 603 in FIG. 6 determines an objectregion from a region that corresponds to the distance within thepredetermined range set by the object distance setting unit 605 or aregion where the reliability by the reliability map is higher than athreshold, and from a candidate region that has been extracted by thematching unit 602. In other words, if the reliability of the distancemap data is smaller than the threshold, processing that performsavoidance so as not to narrow down the candidate region of the matchingresult is performed. According to the present embodiment, using thereliability information representing the reliability of the distanceinformation makes it possible to increase the accuracy of the objecttracking with the use of the correct distance information. Note that, inthe first and second embodiments, although the distance map data isgenerated from the data obtained by the imaging element 141, the presentinvention is not limited thereby. For example, it may be possible togenerate a distance map data map by guiding a part of light that haspassed through the image shooting lens 101 to a range circuit providedseparately from the imaging element 141 by using a half mirror, andobtaining distance information from the range circuit, which is acomponent different from the imaging element 141.

(Other Embodiments)

Embodiment (s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (eg, one or more programs) recorded on a storagemedium (which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment (s) and/or that includes one ormore circuits (eg, application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment (s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment (s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment (s). The computer may comprise one or moreprocessors (eg, central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-097444, filed May 12, 2015, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An object tracking device that obtains imagedata, detects an object, and tracks a target object comprising: one ormore processors; and a memory coupled to the one or more processors andincluding instructions stored thereon that, when executed by the one ormore processors, cause the object tracking device to: collate image dataof a tracking object, which is a tracking target, with image data thathas been obtained, and output information about a candidate region ofthe tracking object; determine an object region relating to the trackingobject from the candidate region that is output; calculate depthinformation of the object; and set a depth range based on informationabout the object region that has been determined and the depthinformation that has been calculated, wherein the object region relatingto the tracking object is determined based on an image region fallingwithin the depth range that is set and the candidate region that isoutput.
 2. The object tracking device according to claim 1, wherein thedepth information is calculated by obtaining data of a plurality ofimages having a parallax and calculating an image deviation amount ofthe plurality of images.
 3. The object tracking device according toclaim 1, wherein the instructions, when executed by the one or moreprocessors, further cause the object tracking device to: calculateinformation about reliability representing the reliability of the depthinformation, wherein an object region relating to the tracking object isdetermined based on an image region having depth information in whichthe reliability that has been calculated is higher than a threshold anda candidate region that is output.
 4. The object tracking deviceaccording to claim 1, wherein the depth range is set to be larger when amoving speed of the tracking object in a shooting direction is fasterthan a predetermined value rather than when the moving speed of thetracking object in a shooting direction is equal to or slower than thepredetermined value.
 5. The object tracking device according to claim 1,wherein information about a depth difference between the tracking objectand an object that is different from the tracking object is obtained,the depth range is set to be larger when the depth difference is largerthan a predetermined value rather than when the depth difference isequal to or smaller than the predetermined value.
 6. The object trackingdevice according to claim 5, wherein, an object that has been determinedto be similar to the tracking object as a result for performing thedetermination of similarity with the tracking object is determined as anobject that is different from the tracking object, and information aboutthe depth difference between the object that has been determined and thetracking object is obtained.
 7. The object tracking device according toclaim 5, wherein a moving speed of an object that is different from thetracking object is determined, the depth range is set to be larger whenthe object is determined to be a stationary object rather than theobject is determined to be a moving object.
 8. A control method executedby an object tracking device that obtains image data, detects an object,and tracks a target object, the control method comprising: collatingimage data of a tracking object, which is a tracking target, with imagedata that is input; outputting information about a candidate region ofthe tracking object; calculating depth information of an object;determining an object region relating to the tracking object from thecandidate region that is output; and setting a depth range based oninformation about the object region that has been determined and thedepth information that has been calculated, wherein the object regionrelating to the tracking object is determined based on an image regionfalling within the depth range that has been set and a candidate regionthat is output.